PSS: Update how commands access backends, so both `backend` and `state_store` configuration can be used #37569

SarahFrench · 2025-09-08T11:55:04Z

Context & how code was structured before this PR

Terraform command implementations obtain an operations backends to use from the (Meta).Backend method. This operations backend may be used for accessing state and/or for running operations.

When using the (Meta).Backend method, calling code may include parsed backend configuration in the BackendOpts argument passed into the Backend method:

// Get the backend config from the root module
backendConfig, configDiags := c.loadBackendConfig(".")
// handle configDiags errors etc

// Pass the config into the Backend method
be, beDiags = c.Backend(&BackendOpts{
	BackendConfig: backendConfig,
	ViewType:      viewType,
})

Or, the calling code could pass a nil value into the Backend method and instead rely on downstream logic to parse any backend blocks in the configuration (this happens if the BackendOpts.BackendConfig value is missing).

// Load the backend
b, backendDiags := c.Backend(nil)

The view type might or might not have been specified in the BackendOpts used. In those cases downstream code would default to using a human view: #37569 (comment)

Changes in this PR

This PR adds a new method: *(m Meta) backend

This method makes it easier to access an instance of an operations backend. This method parses any backend or state_store configuration present in the root module, provides that input to the (Meta).Backend method, and returns the operations backend from there. When pluggable state storage is in use the method takes care of obtaining locks and provider factories necessary for using PSS.

Overall, this PR makes Terraform commands compatible with state stored via state_store. The exceptions are commands like terraform cloud * which expect a specific type of state storage to be present in the configuration when the user executes the command.

Target Release

N/A

Rollback Plan

If a change needs to be reverted, we will roll out an update to the code within 7 days.

Changes to Security Controls

Are there any changes to security controls (access controls, encryption, logging) in this pull request? If so, explain.

CHANGELOG entry

This change is user-facing and I added a changelog entry.
This change is not user-facing.

SarahFrench · 2025-09-08T13:14:27Z

internal/command/show.go

 func (c *ShowCommand) getDataFromCloudPlan(plan *cloudplan.SavedPlanBookmark, redacted bool) (*cloudplan.RemotePlanJSON, error) {
+	mod, diags := c.Meta.loadSingleModule(".")
+	if diags.HasErrors() {
+		return nil, diags.Err()
+	}
+
 	// Set up the backend
-	b, backendDiags := c.Backend(nil)
-	if backendDiags.HasErrors() {
-		return nil, errUnusable(backendDiags.Err(), "cloud plan")
+	b, diags := c.Meta.prepareBackend(mod)
+	if diags.HasErrors() {
+		return nil, errUnusable(diags.Err(), "cloud plan")
 	}


This currently doesn't have any test coverage

internal/command/meta_backend.go

SarahFrench · 2025-10-20T12:45:49Z

internal/command/apply.go

 		}
+		// TODO: Update BackendForLocalPlan to use state storage, and plan to be able to contain State Store config details
 		be, beDiags = c.BackendForLocalPlan(plan.Backend)


I think this is scoped to this ticket: https://hashicorp.atlassian.net/browse/TF-28374
I've linked here from that ticket.

…mmands

…nfig parsing needs to be explicitly added

… can use already parsed config

…eds to be refactored

…ly commands

…in use within operations backend

radeksimko

Some early thoughts after briefly glancing over the diffs

(mostly stylistic) Can we drop all that .Meta stuff from the calling code? Meta is embedded struct inside of every command and it seems unnecessarily verbose to use that notation?
If we now call loadSingleModule() to load the module configuration from various places, doesn't it mean that any diagnostics which aren't directly related to the backend or state store would prevent it from being initialised? 🤔

I can dig deeper and think more the effects and side effects, especially in relation to (2) later.

UPDATE:

Re (2) Upon closer inspection I can now see that the pre-existing logic already calls loadSingleModule and so there is no change in terms of what diagnostics would be raised and when, I suppose. The change just makes this effect more obvious.

radeksimko

Some comments in-line.

Regarding the note about missing test - I think the error case of cloud backend decoding could be covered by a new test in internal/command/meta_backend_test.go?

One more meta-question (no pun intended):

Could we make similar change to the earlier one and drop the .Meta from all of the c.Meta.loadSingleModule?

radeksimko · 2025-10-22T13:46:11Z

internal/command/meta_backend.go

+// This method should be used in NON-init operations only; it's incapable of processing new init command CLI flags used
+// for partial configuration, however it will use the backend state file to use partial configuration from a previous
+// init command.
+func (m *Meta) prepareBackend(root *configs.Module) (backendrun.OperationsBackend, tfdiags.Diagnostics) {


Mostly readability related nitpick: I'd consider making this method's arguments a bit less "greedy", e.g. backend, cloudConfig, stateStore.

Then it makes it more obvious to the reader from all the calling places that this method cannot possibly do any business with the rest of the module.

I could resolve this by making this method load the root module, instead of using data from the calling code.

That would address this concern about duplicated warnings and resembles the original backendConfig method used.

Ah but now I take a proper look I realise that one of the things I was trying to do was avoid the root module being parsed multiple times (e.g. in internal/command/providers.go the diff uses some already-parsed config as the argument to this method).

I'll think on it more.

See #37569 (comment)

internal/command/import.go

internal/command/query.go

internal/command/import.go

radeksimko · 2025-10-22T14:18:11Z

One more note - sorry for the chaos!

I'd say this note in the existing code is relatively important:

terraform/internal/command/meta_config.go

Lines 147 to 151 in 2e5b5de

    
           // Only return error diagnostics at this point. Any warnings will be caught 
        
           // again later and duplicated in the output. 
        
           if diags.HasErrors() { 
        
           	return nil, diags 
        
           }

I'm just unsure how to best preserve it in the wide-scale refactoring here.

…to make relationship to (Meta).Backend more obvious.

… adhere to calling code's viewtype instructions

SarahFrench · 2025-10-30T09:42:53Z

internal/command/meta_backend.go

+	// Only return error diagnostics at this point. Any warnings will be caught
+	// again later and duplicated in the output.
+	root, mDiags := m.loadSingleModule(configPath)
+	if mDiags.HasErrors() {
+		diags = diags.Append(mDiags)
+		return nil, diags
+	}


I've made the new backend method resume being the code to parse the configuration for backend/state_store config. This means that some commands parse the configuration twice, as this code is no longer able to accept parsed config as input.

Ack, I expect this will have some performance impact but also I hope/assume it will be negligible in the grand scheme of things.

Yeah agreed - I think only a few commands would be negatively affected so it's not too bad

internal/command/console.go

…from the backend

radeksimko · 2025-10-31T11:27:42Z

internal/command/graph.go

-	b, backendDiags := c.Backend(&BackendOpts{
-		BackendConfig: backendConfig,
-	})
+	b, backendDiags := c.backend(".", arguments.ViewHuman)


I know this isn't related to your PR really but it's interesting to note the graph command output is always "machine-readable" in the sense that it's DOT language which a human would first pipe into something like graphviz before they can read it. 😅

Though it's not JSON, so neither of the two view types we have are a good match. 🤷🏻

And yes - mimicking the existing default behaviour is a very sensible approach!

radeksimko

What I like about the new changes is how they highlight the differences between various commands and the output formats and maybe make it easier to spot some gaps/opportunities for adding machine-readable output.

radeksimko · 2025-10-31T11:33:23Z

internal/command/output.go


 	// Load the backend
-	b, backendDiags := c.Backend(nil)
+	b, backendDiags := c.backend(".", arguments.ViewHuman)


This command does support both JSON and Human views so I think we should pass the chosen type as an argument instead of hard-coding to ViewHuman here?

Good catch! In that case we may be potentially fixing a bug here; the original code didn't pass a view type in.

internal/command/plan.go

internal/command/providers_schema.go

internal/command/state_meta.go

…ggable state store

… a backend

SarahFrench · 2025-10-31T18:34:06Z

Thanks for that feedback @radeksimko !

When we spoke about testing we discussed using E2E tests on the happy path - I've got that work in draft separate to this PR, and I think the testing for this PR is just that we didn't break usage of backends - which seems to be the case

radeksimko

Since you pointed out the oddity with output command, I looked a bit deeper and ... 😅

I found the primary (only?) reason for us having to pass down the view type is the state locker which is responsible for printing out messages based on that:

terraform/internal/command/views/state_locker.go

Lines 22 to 31 in e44b870

    
           func NewStateLocker(vt arguments.ViewType, view *View) StateLocker { 
        
           	switch vt { 
        
           	case arguments.ViewHuman: 
        
           		return &StateLockerHuman{view: view} 
        
           	case arguments.ViewJSON: 
        
           		return &StateLockerJSON{view: view} 
        
           	default: 
        
           		panic(fmt.Sprintf("unknown view type %v", vt)) 
        
           	} 
        
           }

Some of our commands, like output, graph AFAICT do not perform locking though. It's more because they rely on Backend()/backend() which in turn happens to be also responsible for state migration (?? [*]) and we do locking during state migrations.

[*] How/why we do migration outside of init is a separate question I do not have a conclusive answer to but much of backendFromConfig() pre-dates any PSS related changes so it seems like this existed for some time. 🤷🏻

SarahFrench · 2025-11-03T17:46:31Z

Some of our commands, like output, graph AFAICT do not perform locking though. It's more because they rely on Backend()/backend() which in turn happens to be also responsible for state migration (?? [*]) and we do locking during state migrations.

[*] How/why we do migration outside of init is a separate question I do not have a conclusive answer to but much of backendFromConfig() pre-dates any PSS related changes so it seems like this existed for some time. 🤷🏻

Oh yeah this bit is fun 😆

The Backend() method is used both in init and in non-init commands, which is differentiated by the the Init field in BackendOpts. The Backend() method can detect a difference between config and the backend state file and go on to oversee a state migration if the command is init ((BackendOpts).Init == true) and the necessary flags are set. If a user ran the output command (for example, where (BackendOpts).Init == false) and there was a difference between config and the backend state file then the user would be prompted to run an init command. Logic like here and here uses opts.Init to return early if a change is detected and the user needs to be prompted to run an init command.

I believe that a non-init command would never reach any nitty-gritty logic like state migration due to gating behind opts.Init needing to be true.

In theory we could decide to not set opts.ViewType at all in this PR because backend() is intended to be used for everything except init, but it feels a bit safer to set a value and then allow it to be unused downstream. I think one could argue for either tbh

SarahFrench added the no-changelog-needed Add this to your PR if the change does not require a changelog entry label Sep 8, 2025

SarahFrench commented Sep 8, 2025

View reviewed changes

SarahFrench commented Sep 9, 2025

View reviewed changes

internal/command/meta_backend.go Outdated Show resolved Hide resolved

SarahFrench mentioned this pull request Oct 13, 2025

PSS: Implement initialisation of new working directory (or use of -reconfigure flag) while using state_store #37732

Merged

3 tasks

SarahFrench force-pushed the pss/update-how-commands-access-backend branch 2 times, most recently from f8d4dcc to 57f1d85 Compare October 20, 2025 11:10

SarahFrench changed the base branch from main to pss/init-base-case-no-config-changes October 20, 2025 11:10

SarahFrench force-pushed the pss/update-how-commands-access-backend branch from 57f1d85 to 3df8563 Compare October 20, 2025 12:20

SarahFrench commented Oct 20, 2025

View reviewed changes

Base automatically changed from pss/init-base-case-no-config-changes to main October 20, 2025 15:51

SarahFrench added 12 commits October 20, 2025 16:56

Add a generic method for loading an operations backend in non-init co…

229f42f

…mmands

Refactor commands to use new prepareBackend method: group 1

082f481

Refactor commands to use new prepareBackend method: group 2, where co…

70c4f39

…nfig parsing needs to be explicitly added

Refactor commands to use new prepareBackend method: group 3, where we…

44b9e4a

… can use already parsed config

Additional, more nested, places where logic for accessing backends ne…

5bacdce

…eds to be refactored

Remove duplicated comment

7d5f88d

Add test coverage of (m *Meta) prepareBackend()

323b179

Add TODO related to using plans for backend/state_store config in app…

1afde44

…ly commands

Add testStateStoreMockWithChunkNegotiation test helper

5ba4a6f

Add assertions to tests about the backend (remote-state, local, etc) …

ae6db5c

…in use within operations backend

Stop prepareBackend taking locks as argument

be43ec6

Code comment in prepareBackend

823c5b4

SarahFrench force-pushed the pss/update-how-commands-access-backend branch from 3df8563 to 823c5b4 Compare October 20, 2025 15:57

radeksimko reviewed Oct 22, 2025

View reviewed changes

Replace c.Meta.prepareBackend with c.prepareBackend

96269e9

radeksimko reviewed Oct 22, 2025

View reviewed changes

SarahFrench added 3 commits October 22, 2025 17:38

Change c.Meta.loadSingleModule to c.loadSingleModule

b71931c

Rename (Meta).prepareBackend to (Meta).backend, update godoc comment …

9ca9710

…to make relationship to (Meta).Backend more obvious.

Revert change from config.Module to config.Root.Module

c8b001c

SarahFrench added 2 commits October 24, 2025 18:30

Update (m *Meta) backend method to parse config itself, and also to…

6698109

… adhere to calling code's viewtype instructions

Update all tests and calling code following previous commit

84fb3f0

SarahFrench requested a review from radeksimko October 30, 2025 09:39

SarahFrench marked this pull request as ready for review October 30, 2025 09:39

SarahFrench requested a review from a team as a code owner October 30, 2025 09:39

SarahFrench commented Oct 30, 2025

View reviewed changes

internal/command/console.go Show resolved Hide resolved

SarahFrench added 2 commits October 30, 2025 17:00

Change how an operations backend is obtained by autocomplete code

733618f

Update autocomplete to return nil if no workspace names are returned …

7497796

…from the backend

radeksimko reviewed Oct 31, 2025

View reviewed changes

radeksimko requested changes Oct 31, 2025

View reviewed changes

SarahFrench mentioned this pull request Oct 31, 2025

Update test for autocompletion of top-level commands #37853

Merged

3 tasks

SarahFrench added 4 commits October 31, 2025 17:33

Add test coverage for autocompleting workspace names when using a plu…

0cf0769

…ggable state store

Fix output command: pass view type data to new backend method

b07a903

Fix in plan command: pass correct view type to backend method

99fc261

Fix providers schema command to use correct viewtype when preparing…

e9b15ec

… a backend

SarahFrench requested a review from radeksimko October 31, 2025 18:32

radeksimko approved these changes Nov 3, 2025

View reviewed changes

SarahFrench merged commit f5a28cf into main Nov 3, 2025
9 checks passed

SarahFrench deleted the pss/update-how-commands-access-backend branch November 3, 2025 17:57

	func NewStateLocker(vt arguments.ViewType, view *View) StateLocker {
	switch vt {
	case arguments.ViewHuman:
	return &StateLockerHuman{view: view}
	case arguments.ViewJSON:
	return &StateLockerJSON{view: view}
	default:
	panic(fmt.Sprintf("unknown view type %v", vt))
	}
	}

PSS: Update how commands access backends, so both backend and state_store configuration can be used #37569

PSS: Update how commands access backends, so both backend and state_store configuration can be used #37569

Conversation

SarahFrench commented Sep 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Context & how code was structured before this PR

Changes in this PR

Target Release

Rollback Plan

Changes to Security Controls

CHANGELOG entry

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

radeksimko left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

radeksimko left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SarahFrench Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

radeksimko commented Oct 22, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

radeksimko Oct 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

radeksimko left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SarahFrench commented Oct 31, 2025

Uh oh!

radeksimko left a comment

Choose a reason for hiding this comment

Uh oh!

SarahFrench commented Nov 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

PSS: Update how commands access backends, so both `backend` and `state_store` configuration can be used #37569

PSS: Update how commands access backends, so both `backend` and `state_store` configuration can be used #37569

SarahFrench commented Sep 8, 2025 •

edited

Loading

radeksimko left a comment •

edited

Loading

SarahFrench Oct 30, 2025 •

edited

Loading

radeksimko Oct 31, 2025 •

edited

Loading

SarahFrench commented Nov 3, 2025 •

edited

Loading